feat: use partman and s3 for archival partitions
#610
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maschad
requested review from
Copilot and
jorgeantonio21
and removed request for
jorgeantonio21
There was a problem hiding this comment.
Pull Request Overview
This PR introduces database maintenance enhancements by integrating pg_partman for partition management and S3 for archival and restoration of data. Key changes include:
- New scripts for database maintenance operations (including partition initialization, archival to S3, and restoration).
- Configuration updates in docker-compose and Ofelia for scheduling maintenance tasks.
- Prometheus monitoring integration for the archival process.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/db-maintenance/run-maintenance.sh | Adds a script to execute maintenance tasks, including installing extensions and running SQL scripts. |
| scripts/db-maintenance/restore-from-s3.sql | Implements a PL/pgSQL function to restore data from S3. |
| scripts/db-maintenance/init-partioning.sql | Provides partitioning initialization using pg_partman; note the operator spacing issue. |
| scripts/db-maintenance/archive-partitions.sql | Creates functions and configurations for archiving partitions to S3. |
| scripts/db-maintenance/.env.example | Supplies environment variables required for S3 configuration. |
| prometheus.yml | Configures a new Prometheus job for monitoring the archival process. |
| ofelia.ini | Defines a scheduled job for monthly archive operations. |
| docker-compose.yaml | Updates service definitions to include maintenance tasks and ensure proper volume mappings. |
Comment on lines
45
to
47
| p_template_table = > NULL, | ||
| p_retention = > '36 months', | ||
| p_start_partition = > date_trunc('month', CURRENT_DATE - interval '3 years') :: text |
There was a problem hiding this comment.
There is an extra space in the parameter assignment ('= >' instead of '=>'). This syntax error could prevent the partition maintenance function from executing correctly. Please update to use '=>' consistently.
Suggested change
| p_template_table = > NULL, | |
| p_retention = > '36 months', | |
| p_start_partition = > date_trunc('month', CURRENT_DATE - interval '3 years') :: text | |
| p_template_table => NULL, | |
| p_retention => '36 months', | |
| p_start_partition => date_trunc('month', CURRENT_DATE - interval '3 years') :: text |
Comment on lines
80
to
82
| p_template_table = > NULL, | ||
| p_retention = > '36 months', | ||
| p_start_partition = > date_trunc('month', CURRENT_DATE - interval '3 years') :: text |
There was a problem hiding this comment.
Please remove the extra space in the parameter operator; replace ' = >' with '=>' as required by the function's syntax.
Suggested change
| p_template_table = > NULL, | |
| p_retention = > '36 months', | |
| p_start_partition = > date_trunc('month', CURRENT_DATE - interval '3 years') :: text | |
| p_template_table => NULL, | |
| p_retention => '36 months', | |
| p_start_partition => date_trunc('month', CURRENT_DATE - interval '3 years') :: text |
maschad
changed the title
partman and s3 for archival partitions
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As it stands
reveals
• Total Size: 7314 GB (approximately 7.3 TB)
• Table Size: 584 kB (actual data)
• Index Size: 7314 GB (indexes)
• Estimated Row Count: 1,108 rows
Which shows our indexes are excessively large, whilst this will partly be resolved by #609 to stem the growth, we will still need to clear any bloat that may be stored in TOAST.
This means will need a back up of records as well as going forward once fiat payments are integrated we will also need to store user stack related for the foreseeable future, which means it will need to be archived at intervals to avoid deletion.
This integrates https://github.com/pgpartman/pg_partman to manage archiving via s3